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Course Evaluation in Sweden 
- When, How, What and Why 

Stefan Cronholm 
stefan.cronholm@liu.se 
Dept, of Management and Engineering 
Linkoping University 
Linkoping, 583 81 Sweden 


Abstract 

This study is about course evaluation in Swedish higher education. Performing course evalua¬ 
tion is regulated in Swedish law. Despite this, only half of the courses are evaluated. The aim 
of this study is to understand why satisfactory course evaluations not are performed. Problems 
are identified from a student perspective and the paper provides proposals for reducing the 
problems. In order to tackle the problems an evaluation process model consisting of five 
phases is proposed. A main message is that there is need for an increased government from 
the university's management levels to revise the incentives for performing a satisfactory 
course evaluation. 

Keywords: course evaluation, motivation, evaluation process model 


INTRODUCTION 

The problem we are facing is that course 
evaluation is not performed or is performed 
in an unsatisfactory way. Following prob¬ 
lems are identified: 

• The number of students evaluating 
courses is too small (approx. 35%). 

• The number of courses evaluated is too 
small (approx. 50%). 

• The teachers are not encouraging students 
to perform course evaluation. 

• The students seem to be unmotivated. 

In Sweden, performing course evaluation is 
compulsory. That means it is not optional. It 
is prescribed in the "Regulation of Higher 
Education" (in Swedish: Hogskole- 

forordningen). The regulation reads: "The 
university/college shall offer students possi¬ 
bilities to express their experiences and 
comments through a course valuation that is 
organized by the university/college" (our 
translation), (SFS, 1993). 

The university's action plan for quality work 
includes the following text: "course evalua¬ 
tion shall work as a quality instrument and 


contribute to the quality of the education", 
(Faculty of Arts & Science, 2007). This 
statement makes it clear that course evalua¬ 
tion is part of the university strategy. It is 
amazing that although course evaluation is 
regulated in the law and is part of the uni¬ 
versity's strategy only half of the courses are 
evaluated. The obvious question is why? At 
first sight, course evaluation seems to be 
perceived as something redundant and one 
possible reason is that there are few or no 
incentives for performing the evaluation. 

The aim of this paper is to answer the ques¬ 
tion of why satisfactory course evaluation is 
not performed and to suggest proposals that 
could support the process of evaluation. An 
evaluation process model is proposed. The 
aim of performing course evaluation is obvi¬ 
ous; the results should be viewed as a base 
for a possible redesign of the courses. A 
quote from a student reads "Why should I 
evaluate the course? The results are not 
used anyway". This quote can be true, false 
or something in between. If evaluation re¬ 
sults are used in order to improve courses, 
is there a possibility that the low interest in 
course evaluation could be related to that 
evaluation results and new redesign propos¬ 
als are invisible for students? In other 
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words, is there missing feedback from the 
evaluation? Another related problem is who 
is responsible for organizing possibilities to 
participate in course evaluations? According 
to the "Regulation of Higher Education" the 
Head of School is responsible. In practice, 
this responsibility is delegated to the director 
of study. This makes the question of "why" 
even more interesting since there is no 
doubt that course evaluation is compulsory 
and there is no doubt where the responsibil¬ 
ity lies. 

This paper takes a student perspective since 
it is the students who are performing the 
evaluations and it is the students who are 
primarily benefiting from the results. Taking 
a student perspective means that the stu¬ 
dent's opinions are identified and analyzed. 
Course evaluation is viewed as an important 
instrument for students to preserve their 
influence on how their education is designed. 
The character of the knowledge searched 
can be seen as both explanative and norma¬ 
tive. The aim of the question "Why is course 
evaluation not performed?" is to identify ex¬ 
planative knowledge while the aim of the 
proposals is of normative character. The 
evaluation process of the subject information 
systems has been studied at one Swedish 
university. Possible generalisations will be 
discussed in the concluding section. 

This introductory section is followed by sec¬ 
tion 2; the research approach. Section 3 
describes the current evaluation process and 
section 4 contains a discussion about rele¬ 
vant theories. After that the findings will be 
presented in section 5 and finally conclu¬ 
sions will be drawn in section 6. 

2. RESEARCH APPROACH 

The research approach can be characterized 
as abductive (Peirce, 1931-35; Alvesson & 
Skoldberg, 1999) and as qualitative (Glaser 
& Strauss, 1967; Strauss & Corbin, 1998; 
Strauss & Corbin, 2007). The concept abduc¬ 
tive means that the research process has 
been both inductive and deductive. Inductive 
in the sense that problems concerning 
course evaluation have been induced from 
empirical data. Deductive in the sense that 
existing theory has been used for compari¬ 
son of induced data and as inspiration of 
generating proposals for problem reduction. 
There has been a continuous shift between 


empirical data and existing relevant theories 
(see figure 1). 


O 








Empiri cal 
data 

Figure 1. The research process 

In order to answer the question asked data 
has been gathered through interviews of 
students, studies of reports concerning 
course evaluation (Stake, 1993) and general 
theories about evaluation (Scriven , 1967; 
Scriven , 1972; Rutman, 1980; Remenyi & 
Sherwood-Smith, 1999; Love, 1991; Wal- 
sham, 1993). All together there have been 
six interviews conducted with students rep¬ 
resenting different study levels and genders. 
The interviews were constructed as semi- 
structured (Patton, 1990). That is, the 
character of the interviews was more like a 
conversation than formally structured inter¬ 
views (Patton, 1990). That means, a prede¬ 
fined order of questions have not been fol¬ 
lowed, rather there have been an openness 
and sensitivity for the students' opinions. 

Based on the results of the interviews, sev¬ 
eral problems have been identified. The 
problems have been related to each other in 
terms of cause and effect by using problem 
diagrams (Goldkuhl & Rostlinger, 2003). Fi¬ 
nally, an evaluation model is proposed in 
order to reduce the problems identified. The 
proposals are based on the interviews with 
the students and theories about evaluation. 
The deficiencies in current course evaluation 
approaches could be seen as anomalies that 
need attention. In this way the knowledge 
contribution can be viewed as cumulative, 
that is, the evaluation process model devel- 
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oped is built on existing knowledge/theory 
where the 'good' parts are preserved and 
new proposals are added. 

We have identified four groups that could 
benefit from the results of this paper. The 
first group is of course the students them¬ 
selves. The students want to take courses 
that are of high quality. The second group is 
the program managers. A program manager 
is responsible for a study program and has 
thereby an interest in the quality of courses. 
The third group is the teachers. Outcome of 
course evaluations is normally excellent 
feedback for improving and developing 
courses. The fourth group is the university 
level. The university is legally responsible for 
how course evaluations are performed. A 
working course evaluation process with an 
acceptable number of course evaluators se¬ 
cures that the university is fulfilling the re¬ 
quirements according to the regulation. 

3. BRIEF DESCRIPTION OF THE 
CURRENT EVALUATION PROCESS 

The evaluation process consists of two phas¬ 
es, one phase where a questionnaire is con¬ 
structed by the teacher and one phase 
where questions are answered by the stu¬ 
dents. The questionnaire has to embrace 10 
compulsory questions provided by the prin¬ 
ciple. These questions are standard ques¬ 
tions and are used for comparisons between 
courses. In addition to the principle's ques¬ 
tions the teacher adds an arbitrary number 
of course specific questions. A questionnaire 
normally consists of 40-50 questions. Exam¬ 
ples of questions are: is the aim of the 
course fulfilled, did you get informative 
course information, did you benefit from the 
course literature, are you happy with the 
teachers' pedagogy, did the examination 
correspond to the content of the course and 
how much effort did you put in. Answers to 
questions can be expressed both qualitative¬ 
ly and quantitatively. 

When the questionnaire is constructed, the 
students receive an e-mail that the ques¬ 
tionnaire is electronically available and after 
a week a reminder is sent out. The process 
of construction and answering is completely 
computer supported. Using a computer sup¬ 
ported evaluation process is a directive from 
the principle. Of course, paper based evalua¬ 
tion can be used but only as a complement. 
Still, paper based evaluations are used as 


the only medium. In that case, the teacher 
provides the students with a paper based 
questionnaire at the end of course. Normally 
the questionnaire is distributed at the end of 
the last lecture. There is also the case that 
course evaluation is not performed at all. 

4. RELATED THEORIES 

Course evaluation is one type of evaluation; 
therefore it would be possible to benefit from 
general theories about evaluation. Evalua¬ 
tion is defined as "collection and use of in¬ 
formation to make decisions about educa¬ 
tional programs", (Cronbach, 1963). Accord¬ 
ing to Cronbach et al. (1980) evaluation 
should be recognized as not only a service to 
central decision makers, but should help 
everyone in a pluralistic society understand 
what programs accomplish and why they fall 
short of their objectives. These claims for¬ 
mulated as general statements about 
evaluation are valid for the concept of 
course evaluation. 

According to several theories about evalua¬ 
tion the term motivation seems to be a key 
concept. That is, people who are doing 
evaluation have to be motivated; they need 
incentives for participating in evaluations. 
One of the best incentives is that there is 
visible correlation between the evaluators' 
opinions and the actions taken. According to 
Heron & Reason (2001) people who are af¬ 
fected by changes should always have the 
opportunity to participate in evaluation or 
change work. Dwyer (2008) claims that mo¬ 
tivating people is a myth and furthermore 
that people cannot be motivated by others. 
The claim is that motivation comes from 
within and leaders can only provide condi¬ 
tions for motivation like setting up an envi¬ 
ronment that stimulates the motivation. The 
question "What's in it for me?" is critical for 
individuals in order to be motivated (ibid.). 
This question is clearly valid concerning the 
students' motivation for performing course 
evaluation. 

Maslow's (1943) conceptualization of a 
hierarchy of human needs is often used as a 
base for understanding motives. The 
hierarchy consists of the levels physiological 
needs, safety needs, belonging needs, es¬ 
teem needs and self actualisation need. 
Physiological needs are the very basic needs 
such as air, water, food, sleep, shelter, etc. 
Safety needs have to do with personal safety 
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and security including job security. Belong¬ 
ingness is the desire to belong to groups: 
clubs, work groups, religious groups, family, 
gangs, etc. There are two types of esteem 
needs. First is self-esteem which results 
from competence or mastery of a task. Sec¬ 
ond, there's the attention and recognition 
that comes from others. The need for self- 
actualization is "the desire to become every¬ 
thing that one is capable of becoming". The 
aim of course evaluation is to provide input 
for improving the study environment and 
content of courses. This can be seen as a 
need that exists on the higher levels; es¬ 
teem and self actualisation. 

Herzberg et al. (1959) is discussing motiva¬ 
tion in terms of hygiene factors. They claim 
that if all the hygiene factors are taken care 
of, you have created an environment that 
motivates people. In other words, the hy¬ 
giene factors stop people from being unmo¬ 
tivated. The hygiene factors included in a job 
environment encompass the company, its 
policies and its administration, the kind of 
supervision which people receive while on 
the job, working conditions, interpersonal 
relations, salary, status and security. Ac¬ 
cording to Herzberg et al. (1959), these fac¬ 
tors do not lead to higher levels of motiva¬ 
tion but without them there will be dissatis¬ 
faction. The hygiene factors mentioned could 
easily be transformed into the context of 
course evaluation. 

A well known "prescription" to increase mo¬ 
tivation is to provide feedback. Feedback 
means to inform people about the conse¬ 
quences of their actions. The aim of provid¬ 
ing feedback is to inform people of how they 
are doing in relation to a specific goal 
Stephen (2002). Moreover, it is important to 
provide a feedback that is timely, effective 
and appropriate. Feedback should be deliv¬ 
ered in a way that doesn't make people de¬ 
fensive. Rather, feedback should engage 
people to perform actions leading to specific 
goals. When is it timely? When is it appro¬ 
priate to give feedback? Stephen (2002) 
claims that feedback could be given at any¬ 
time you want to improve a performance. 

According to Deci (1992) true motivation is 
based on a genuine interest. Deci claims that 
there are three levels that affect peoples' 
behaviour; 1) embedded regulation; that is, 
the only reason for performing the task is 
that you have to do it. For example a 


teacher demands that a student performs a 
task. In this level there is no free will in¬ 
volved, the student is governed by the 
teacher's reactions. Level 2 two is about 
identified regulation; that is, you are not 
longer thinking of other peoples' demands. 
The demands are adopted as your own de¬ 
mands. The tasks are thereby performed as 
free will and not seen as coercion. The third 
level is called integrated regulation; that is, 
the demands are incorporated with your ex¬ 
isting value system. If external demands are 
in line with existing values, the motivation 
for performing a task will be reinforced. 

All the theories discussed above claim that it 
is hard to develop an environment that im¬ 
proves peoples' motivation. The reason is 
that there is no "one size fits all" solution to 
the problems since every course evaluator 
(student) is an individual. In order to in¬ 
crease the students' motivation it seems im¬ 
portant to identify existing problems and 
identify incentives for increasing the motiva¬ 
tion. 

5. FINDINGS - AIMS AND 
PROBLEMS 

Aims identified - Why course 
evaluation? 

Three different aims have been identified. 
The aims are related to three significant 
roles that are involved in ordering, giving 
and taking courses. The three roles are pro¬ 
gram managers (faculty), study direc¬ 
tor/teachers (department) and students. 

Aim 1 - Provide an instrument for presenting 
experiences for courses taken. The first aim 
is related to the students; the course par¬ 
ticipants. The aim is to provide an instru¬ 
ment for students to present their opinions 
and experiences about courses taken. This 
opportunity is a right of co-determination 
and regulated in Swedish law. The instru¬ 
ment is used for presenting positive or nega¬ 
tive opinions. Moreover, several students 
prefer to view the relation between the uni¬ 
versity and themselves as a business rela¬ 
tion. The university is the supplier of knowl¬ 
edge and the students are the customers. 
Therefore a course is viewed as a product (a 
package of knowledge). Through using the 
instrument course evaluation, students can 
present opinions that improve the quality of 
the product. 
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Aim 2 - Provide feedback to teachers and 
study directors (course suppliers). The sec¬ 
ond aim is related to the course suppliers. 
The aim is to provide feedback to teachers 
and study directors that can be used for im¬ 
plementing course improvement. The ques¬ 
tion "how did the students perceive the 
course?" is important for teachers in order to 
propose improvements. 

Aim 3 - Provide feedback to the university 
administration. The third aim is to provide 
feedback to program managers and the fac¬ 
ulty level. At the studied university programs 
managers are ordering and paying for 
courses delivered from study directors de¬ 
partment. The program director is interested 
in results from course evaluations in order to 
see if he got what he paid for. 

Problems identified - How, When 
and What 

All together nine problems have been identi¬ 
fied. The problems are of different character 
but they are all related to the conditions of 
the evaluation process, i.e. to the evaluation 
process itself or to the results of the evalua¬ 
tion process. 

Problem 1 - The regulation is not detailed 
enough. According to the students, courses 
are in several cases evaluated in an unac¬ 
ceptable way or not evaluated at all. The 
students think that it is good that course 
evaluation is regulated in the law, but they 
would like the law to be more precise and 
detailed about how course evaluation ought 
to be performed. They perceive the law text 
as being too general and thereby there are 
too many action possibilities of how to con¬ 
duct course evaluation. 

Problem 2 - Insufficient feedback. Most of 
the students think that the feedback is insuf¬ 
ficient. They are not satisfied with informa¬ 
tion that is too aggregated. The students 
also think that the results of the evaluation 
are hard to find. 

Problem 3 - Low engagement by students. 
The results of the interviews divide the stu¬ 
dents into three categories. First, there are 
engaged students that present opinions in 
order to improve courses. Second, there are 
students that are filling in the evaluation 
form as a matter of routine and in an unre- 
flective way. Third, there a students who 
are not interested at all and don't participate 


in course evaluation. Several students seem 
to be aware that course evaluation is some¬ 
thing that should be done, but far from all 
are aware of why. The students blame lazi¬ 
ness for not filling in the questionnaire. 

Problem 4 - Performing course evaluation is 
too time consuming. The students think that 
it takes a long time to answer the question¬ 
naire. There are too many questions and 
several of the questions seem to be unim¬ 
portant. The reason why there are a large 
number of questions is that the answers are 
used on different levels of the hierarchy of 
the university. There are questions asked by 
the university level, there are questions 
asked by the faculty level and there are 
questions asked on the course level. All to¬ 
gether there can be over 50 questions 
asked. 

Problem 5 - Teachers are not receptive to 
criticism. Several students are questioning if 
course evaluation will have an impact on 
courses at all. These students perceived 
some of the teachers being not receptive to 
criticism. Despite that negative comments 
were brought forward there were no visible 
reactions from teachers. This behaviour 
made the students upset and reinforced 
their understanding that course evaluation is 
meaningless. 

Problem 6 - The students will not benefit 
from the results. The students' understand¬ 
ing is that the result of the evaluation will 
not influence their education. They have al¬ 
ready finished the course so why should 
they care. The argument that they should 
participate in course evaluation since their 
precedents did that for their sake is not 
powerful enough. 

Problem 7 - The questionnaires are not dis¬ 
tributed adjacent to the course end. Some¬ 
times questionnaires are distributed weeks 
or even months after a course is finished. 
There is a risk that the students forget their 
experiences or that their engagement is low 
when a long time has passed between the 
course end and the evaluation. 

Problem 8 - Measures to take corresponding 
to the identified problems are not developed. 
The identified problems are not always 
transferred into measures to take. Problems 
have been identified but it happens that no 
improvements are implemented. It seems 
like the teachers are in charge of the process 


© 2010 EDSIG 


http://isedj.org/8/16/ 


April 26, 2010 



ISEDJ 8 (16) 


Cronholm 


of deciding if a problem should be taken care 
of or not. The students are not participating 
in this process and thereby this part of the 
course evolution process is out of their con¬ 
trol. 

Problem 9 - The outcome of course evalua¬ 
tions is not easily accessible. The students 
claim that previous course evaluations are 
not easy to access. They could be stored in 
the teacher's local computer, in the teacher's 
bookshelf or in a central IT-system. If stu¬ 
dents are not informed of the results of the 
course evaluations their motivation for doing 
course evaluation will decrease. 

Problem relations 

In order to be able to suggest proposals to 
the problems identified a more thorough 
problems analysis is needed. According to 
Goldkuhl & Rostlinger (2003) problem analy¬ 
sis is an organizational problem solving 
process. Problem analysis is a process of 
defining problems and proposing solutions. 
This will not automatically lead to problem 
resolution (ibid.). Besides identified prob¬ 
lems and proposing solutions change meas¬ 
ures must be developed and implemented. 
Moreover, it can be the case that not all 
problems are possible to eliminate. There 
are situations, which you must keep up with. 
Hopefully, there is a way to reduce the prob¬ 
lems when you can't eliminate them. One 
way to more thoroughly analyze problems is 
to use Change Analysis (Goldkuhl & Rostlin- 
ger, 2003). Change Analysis consist of a 
method component called problem diagrams 
which aims to analyze problems in terms of 
cause and effect. This way of analyzing 
problems is also supported by Strauss & 
Corbin (1998, 2007). 

In the problem diagram below an analysis of 
problem causes and problem effects are do¬ 
cumented. In order to support the reading of 
the diagram some comments are provided: 
The problem "PI Regulation is not detailed 
enough" is seen as a problem that causes 
several problem effects. The problem "P8 
Problems are not transferred into measures 
to take" is viewed both as a problem cause 
end a problem effect. P8 is part of causing 
"P2 Insufficient feedback" and is also an ef¬ 
fect since it is caused by "P5 Teachers are 
not receptive". Problem "P3 Low engage¬ 
ment of students" is seen as the ultimate 
problem effect. 



Figure 2. Relations between identified 
problems 

Problem reduction 


The proposals recommended in order to re¬ 
duce the problems are based on the student 
interviews and on existing theories of 
evaluation and motivation (see section 4). In 
this way the proposals suggested in order to 
reduce the problems are grounded in both 
empirical findings and in existing theory. 

Problem 1 - The regulation is not detailed 
enough. 

Proposal: Inform the university administra¬ 
tion that the students wish more detailed 
regulations. Encourage the university ad¬ 
ministration to identify if other (contradic¬ 
tory) opinions exist among other stake hold¬ 
ers (teachers, study directors). 

These students think that there is too much 
freedom and flexibility for the universities to 
plan and conduct course evaluation. In other 
words there is a need for an increased guid¬ 
ance and added detailed recommendations 
for how course evaluation ought to be per¬ 
formed. The Regulation of Higher Education 
should be seen as a framework (SFS, 1993). 
There is no hindrance for the universities to 
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add detailed recommendations as long as 
they fit into the framework. The students 
support that course evaluation is regulated 
in the law, but they would like the law to be 
more precise about how course evaluation 
ought to be performed. There could be other 
opinions among other stake holders (teach¬ 
ers, study directors, administration staff) 
who prefer regulations to be more openly 
formulated. As a complement to this study 
there is a need to identify if other prefer¬ 
ences exist in order to gain a richer base for 
decision. 

Problem 2 - Insufficient feedback. 

Proposal - Make feedback compulsory; com¬ 
pensate teachers. 

There is no (economical) incentive for teach¬ 
ers to provide feedback to students. Several 
teachers are doing as little as possible since 
they are fully loaded with regular teaching. 
In order to get rid of this problem, teachers 
must be compensated for both offering 
course evaluation and providing feedback to 
students. Providing feedback should not be 
an option; it should be compulsory. Accord¬ 
ing to Nielsen (1993) feedback should be 
provided to people within reasonable time. 

Problem 3 - Low engagement of students. 

Proposal - Inform the students of why 
course evaluation is important. 

First, all students should be informed about 
why course evaluation is important. There 
should be no doubt why this opportunity 
could improve the education. According to 
Moxner (1984) students could be aware of 
the importance of course evaluation. Despite 
this, students chose not to perform evalua¬ 
tions. Moxner claim that every person has a 
"need of comfort". Engaging in course 
evaluation could mean that students have to 
leave the social and secure environment. 
Presenting opinions in a course evaluation 
could result in an imbalance of the "need of 
comfort". 

Second, the engagement of the students will 
increase when they experience concrete re¬ 
sults from performing course evaluation. 
The current situation can be compared to a 
"bad circle". That is, no visible changes of 
courses will probably lead to a low degree of 
engagement. A low degree of engagement 
will not create "good" proposals for course 
improvements. The idea is to move from a 


"bad circle" to a "good circle". The imple¬ 
mentation of course improvements must be 
made visible for the students. They should 
be marketed and students should be in¬ 
formed about improvements made. If the 
students experience that teachers and study 
directors are listening to their opinions; it 
will increase their engagement and their mo¬ 
tivation. This is in line with claims by Heron 
& Reason (2001). They claim that a close 
dialogue and a broad participation is the key 
to increase the engagement. Of course, 
there will never be 100% of engaged stu¬ 
dents, but all students should at least be 
aware of when, where, how and why course 
evaluation is performed. 

Problem 4 - Performing course evaluation is 
too time-consuming. 

Proposal - Reduce the number of questions 
asked. 

The students think that it takes a long time 
to answer the questionnaire. According to 
the students, an objective should be that the 
minutes spent on course evaluation should 
be equal to the number of credits of the 
course. That is, a student taking a course 
embracing 10 credits should spend a maxi¬ 
mum of 10 minutes on the course evalua¬ 
tion. The analysis of why there are so many 
questions asked for each course shows that 
there are three different questioners; the 
university, the faculty and the teacher. The 
data gathering from the university levels and 
the faculty level seems to be too ambitious. 
Their interest is to gather data of a more 
general nature and not at the detailed 
course level. Therefore it is not necessary 
that questions asked by the university and 
by the faculty have to be done at every 
course evaluation. It should be sufficient to 
gather these data once or twice every se¬ 
mester. We are not saying that the ques¬ 
tions asked by the university and the faculty 
level are unimportant; rather our claim is 
that the frequency of data gathering is ex¬ 
cessive. Following this proposal means that 
the number of question can be substantially 
reduced. 

Problem 5 - Teachers are not receptive to 
criticism. 

Proposal - Make course improvements 
transparent for students and follow up on 
the implementation of proposals for course 
improvements. 
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Several students are questioning if results 
from course evaluation will improve the 
courses at all. They mean that some of the 
teachers are not receptive to criticism. A 
closer look revealed that most of the courses 
are improved based on course evaluations. 
The problem seems to be that the improve¬ 
ments are not made visible to the students. 
Therefore this problem is overlapping the 
problem of feedback (problem 2). For a mi¬ 
nor part of the courses proposals have not 
been implemented. In these cases, there 
must be a follow-up from study directors. 

Problem 6 - The evaluating student will not 
benefit from the results. 

Proposal - Clarify that the evaluating stu¬ 
dent can benefit from the results of the 
evaluation 

A common misunderstanding is that the re¬ 
sult of course evaluation only affects the 
next version of the course. A course evalua¬ 
tion normally contains questions about the 
content, the teacher's pedagogical skill and 
the way the course is organized. That means 
that comments from students could be 
viewed as course specific or being of a more 
general nature. Specific comments about the 
course will hopefully have an impact on the 
next version of the course; but the general 
comments will also have an impact on other 
courses where the same teacher is involved. 
A teacher who is sensitive of criticism will 
remove bad ideas and replace them with 
something new in all courses he is involved 
in. Often students will meet the same 
teacher again in another course at the same 
or at a higher level. 

Course evaluation is traditionally performed 
after the course is finished. Another possibil¬ 
ity to make students benefit from the ongo¬ 
ing course is to offer an opportunity to 
evaluate while the course is running through 
using muddy cards. An evaluation opportu¬ 
nity that is offered in the middle of the 
course will increase the motivation of the 
students (Kessler & Nadim-Tehrani, 2002). 

Problem 7 - The questionnaires are not dis¬ 
tributed adjacent to the course end. 

Proposal - Make both teachers and students 
aware of problems related to this problem 
and why this is important to follow. 

This problem is of an administrative charac¬ 
ter and should be easy to get rid of. There 


should be a clear and agreed time limit for 
when course evaluations should be distrib¬ 
uted. 

Problem 8 - Measures to take corresponding 
to the identified problems are not developed. 

Proposal - Invite student representatives to 
discuss improvements and use written 
agreements. 

Students can be invited to discuss which 
measures to take corresponding to the prob¬ 
lems identified. Open up this part of the de¬ 
cision process in a democratic way is one 
way to increase the students' motivation. We 
are not saying that the decisions should be 
made in a democratic way. The decisions 
should always be made by the university 
since they have to take responsibility for the 
consequences. We are proposing that the 
students can provide input to the decision 
process through creative ideas about how 
courses could be improved. They possess a 
unique knowledge as course participants; 
not investigating their knowledge is a lost 
opportunity. Discussing different opportuni¬ 
ties for improvement will also increase the 
students understanding for why some pro¬ 
posals of improvement are less feasible and 
why others are more feasible. Advocates of 
a close collaboration argue for a broad and 
genuine participation aiming at agreement 
(Heron & Reason (2001) and Carlshamre 
(1994). 

We are also proposing that the outcome of 
the discussion is documented. A written 
document can be seen as a memory of the 
decisions made. This document should then 
be handed over to the students of the next 
year. The aim of the document is to serve as 
a basis for follow up activities. 

Problem 9 - The outcome of course evalua¬ 
tions is not easy to access. 

Proposal - Make outcomes of course evalua¬ 
tions available from the web site. 

The students claim that previous course 
evaluations are not easy to access. They are 
also self-critical and admit that they could 
put more effort in searching for course 
evaluations. Nowadays almost every course 
has a web site where information about the 
course can be found. A simple proposal to 
reduce this problem is to make the outcome 
of the course evaluation available from this 
web site. Together with the presentation of 
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the results newly implemented improve¬ 
ments could be highlighted. Simple actions 
like this will make the improvements visible 
for the students, provide feedback on course 
evaluation and contribute to more engaged 
students. 

The evaluation model 

The problems identified can be categorized 
as belonging to the management level and 
to the operational level. The management 
level category consists of problems that 
need to be taken care of on a higher hierar¬ 
chical level and are outside the scope of a 
traditional course evaluation process. The 
operational category consists of problems 
that can be taken care of within the course 
evaluation process. The two categories are 
not mutually exclusive since they are over¬ 
lapping each other. That is, some problems 
exist in both categories but different aspects 
of the problems are focused. 

The management level orientated problems 
are "PI The regulation is not detailed 
enough", "P2 Insufficient feedback (the 
management level needs to make sure that 
teachers will provide students with feedback 
and make sure that there exists an incen¬ 
tive), ”P6 The evaluating student will not 
benefit from the results" (the management 
level needs to inform and market that stu¬ 
dents will benefit from participating in 
course evaluations), "P8 The identified prob¬ 
lems are not transferred into measures to 
take" (the management level needs to fol¬ 
low-up that good proposals for improve¬ 
ments are taken care of") and "P9 The out¬ 
come of course evaluations are not easily 
accessible (the management level needs to 
inform teachers of how, when and why re¬ 
sults should be presented. The operational 
level is divided into five phases: follow-up 
meeting, present improvements, mid course 
evaluation (Kessler & Nadim-Tehrani, 2002), 
quantitative evaluation (Gummesson, 1988; 
1970: Patton (1990) and qualitative evalua¬ 
tion (Kvale, 1989; Bryman, 2001; Strauss & 
Corbin 2007). 

The first phase is a follow-up meeting. The 
follow-up meeting should be organized ap¬ 
proximately one month before commence¬ 
ment of the course. The term follow-up is 
used since the aim is to follow up or to check 
whether the agreed and documented im¬ 
provements from the previous version of the 


course have been implemented or not. A 
written protocol from the qualitative evalua¬ 
tion (see phase five) has been handed over 
to this year's students. The student repre¬ 
sentatives ask for a meeting with the re¬ 
sponsible teacher. At the meeting, the 
teacher and the student representatives 
walk through the protocol. This phase works 
more or less as a checkpoint where the 
teacher can verify that agreed improve¬ 
ments are implemented or present argu¬ 
ments for why they are not. Presenting ar¬ 
guments for why an improvement proposal 
has not been implemented is also feed back 
that is important to bring forward. This 
phase addresses primarily the problems "P2 
Insufficient feedback", "P3 The engagement 
of the students", "P5 Teachers are not re¬ 
ceptive for criticism", "P8 The identified 
problems are not transferred into measures 
to take" and "P9 - The outcome of course 
evaluations are not easily accessible". This 
phase addresses all the three aims (see sec¬ 
tion 5.1). 

Phase two, presenting improvements, aim¬ 
ing at giving the students a clear feedback of 
what course improvements have been im¬ 
plemented since the last time the course 
was running. This information should be pre¬ 
sented at the first lecture in front of all the 
students. In that way the students will be 
aware that their experiences from participat¬ 
ing in courses will affect the course content 
or the pedagogy used. This awareness will 
increase their motivation and attitudes to¬ 
wards course evaluation. Phase two ad¬ 
dresses all the three aims (see section 5.1) 
and addresses primarily the problems "P2 
Insufficient feedback", "P3 The engagement 
of the students", "P5 Teachers are not re¬ 
ceptive to criticism", "P8 The identified prob¬ 
lems are not transferred into measures to 
take" and "P9 The outcome of course 
evaluations is not easy to access". 

The aim of the third phase, mid course 
evaluation, is to offer a possibility for the 
students to influence the running course. 
Being able to present opinions about the 
running course will motivate the students for 
performing evaluation since the result will 
have an affect on the remaining part of the 
running course and not primarily on the next 
version of the course. This phase addresses 
primarily the problems "P2 Insufficient feed¬ 
back", "P3 The engagement of the students", 
"P5 Teachers are not receptive to criticism". 


© 2010 EDSIG 


http://isedj.org/8/16/ 


April 26, 2010 



ISEDJ 8 (16) 


Cronholm 


12 


"P6 The evaluating student will not benefit 
from the results" and "P8 The identified 
problems are not transferred into measures 
to take". This phase addresses all the three 
aims (see section 5.1). 

The fourth phase, quantitative evaluation, 
addresses the first aim of providing an in¬ 
strument for all students to present their 
opinions and experiences. A questionnaire 
consisting of structured and predefined 
questions is sent to all the students. An¬ 
swers could be given according to predefined 
options or as free text. The advantage of 
starting with a qualitative survey is that data 
can be gathered on a broad array (Sears, 
1997). The answers are compiled according 
to general statistical methods. This phase 
primarily addresses the problems "P2 Insuf¬ 
ficient feedback", "P3 The engagement of 
the students", P4 Performing course evalua¬ 
tion is too time consuming" and ”P7 The 
questionnaires are not distributed adjacent 
to the course end". 


Management of course evaluation 


I 



Figure 3 The course evaluation model 

The fifth phase consists of a qualitative 
evaluation approach. This phase addresses 
all the three aims (see section 5.1). The idea 
is to use the results from phase four (the 
quantitative evaluation) and to gather more 
data around problems that are of more con¬ 
cern. The major strengths and problems are 
focused. Student representatives, teachers 
and study directors are meeting face-to- 
face. The aim of this meeting is to identify 
and suggest proposals for improvements. A 
protocol is used in order to document what 


has been agreed upon. Of course, it can be 
the case that teachers and students could 
not reach an agreement. In this case the 
teacher should provide clear arguments of 
why a proposed improvement is neglected. 
Otherwise there is a risk that the students 
perceive the teacher is not being receptive 
to criticism. This phase addresses primarily 
the problems "P2 Insufficient feedback", "P5 
Teachers are not receptive for criticism" and 
"P3 The engagement of the students". 

6. CONCLUSIONS 

This study has revealed a number of prob¬ 
lems concerning course evaluation. Some 
problems might be well known while others 
might be unknown which could depend on 
different contexts or cultures. The knowl¬ 
edge contributions of this paper are: 

• a documented and structured analysis of 
problems and problem relations that 
could appear in the context of course 
evaluation (see section 5.2-5.3) 

• proposals for reducing the problems (see 
section 5.4) 

• an evaluation process model (see section 
5.5) 

The aim of course evaluation is to provide 
quality. Today, quality is the most important 
competition factor among universities. 
Therefore course evaluation should be used 
as one of the most important instruments to 
improve education. Course evaluation should 
not be perceived as a burden or something 
that you are forced to do. Perceiving course 
evaluation as a burden can be compared to 
Deci's three levels of behaviour (1992), (see 
section 4). Deci call one level for "embedded 
regulation", that is, the only reason for per¬ 
forming the task is that you have to do it. In 
this case there is no free will involved. Ac¬ 
cording to Deci the motivation will increase if 
the behaviour is in line with what Deci calls 
"integrated regulation". That is, the de¬ 
mands are incorporated with your existing 
value system. Perceiving course evaluation 
as "integrated regulation" means a change 
of attitude towards course evaluation; to see 
course evaluation as an excellent quality 
instrument and not as a burden. 

The main message in this study is that the 
questions why, what, when and how must 
be considered when planning for course 
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evaluation. To plan means to organize a 
process that is more structured and formal 
than the character of the current evaluation 
process. Too much government concerning 
proposals of structures and detailed proc¬ 
esses could be perceived as offensive, espe¬ 
cially within the academy where teachers 
often are used to possessing a high degree 
of freedom of action. On the other hand, too 
much freedom could lead to a situation 
where course evaluations are perceived as 
optional. The findings in this study show that 
there is need for an increased governing and 
the establishment of incentives for perform¬ 
ing a satisfactory course evaluation. 

Two established concepts discussed in gen¬ 
eral evaluation theory are goal-based 
evaluation and goal-free evaluation. Goal- 
based evaluation is defined as measuring the 
extent to which a program or intervention 
has attained clear and specific objectives 
Patton (1990) and goal-free evaluation is 
defined as gathering data on a broad array 
of actual effects and evaluating the impor¬ 
tance of these effects in meeting demon¬ 
strated needs (Patton, 1990, Scriven, 1972). 

The proposed course evaluation can be cha¬ 
racterized as being both goal-based and 
goal-free. For example, phase four "qualita¬ 
tive course evaluation" is primarily a goal- 
based activity since the aim is to measure if 
predefined goals are fulfilled or not; to what 
extent and in what ways. Phase 5 "qualita¬ 
tive course evaluation" is both goal-based 
and goal-free. Besides discussing the out¬ 
come of the phase 4 other issues not cov¬ 
ered by the questionnaire used in phase 4 
are also discussed. 

Other established concepts in general evalu¬ 
ation theory are formative and summative 
evaluation. The aim of formative evaluation 
is to provide a systematic feedback to the 
designers and implementers during an ongo¬ 
ing development process (Walsham, 1993; 
Scriven, 1967). Summative evaluation is 
concerned with identifying and assessing the 
worth of programme outcomes in the light of 
initially specified success criteria after the 
implementation of the change programme is 
completed (Scriven, 1967). The proposed 
evaluation embraces both formative and 
summative evaluation. Phase 3 "mid-course 
evaluation" is primarily formative while 
phase 4 "quantitative course evaluation" and 


phase 5 "qualitative course evaluation" are 
primarily summative. 

This study can be criticized for being prob¬ 
lem oriented and not considering existing 
strengths. In parallel with being problem 
oriented there is also a need of identifying 
strengths that needs to be preserved. The 
risk of only being problem oriented is that 
you can "throw out the baby with the bath¬ 
water". Looking closer into strengths is 
therefore proposed as a direction for future 
research. 

This study can also be criticized for propos¬ 
ing a too ambitious evaluation process. En¬ 
couraging students to participate in different 
evaluation phases over and over again could 
lead to "tiredness of evaluation". According 
to The Swedish National Agency for Higher 
Education (2003) the advantages with pre¬ 
senting a model that creates conditions for 
increasing the motivation should exceed the 
disadvantages. According to White (1959) 
the competence is an important factor for 
increasing motivation. This claim is also ap¬ 
plicable in the context of course evaluation. 
The results from course evaluation should 
lead to improvements of the course content. 
Updated course content will of course create 
a good condition for improving the compe¬ 
tence of the students. 

Another limitation in this study is that the 
results only are based on the student per¬ 
spective. Other complimentary perspectives 
such as the teachers', the study directors' 
and the university levels' perspectives would 
probably bring forward other problems that 
might be in conflict with the students' inter¬ 
ests. In this study the students' perspective 
is chosen. The reason is that it is the stu¬ 
dents who will primarily benefit from course 
evaluations. 

The scope of this study is limited to Swedish 
conditions for performing course evaluation 
within the subject of information systems. A 
reasonable question to ask is: are the find¬ 
ings valid for other conditions as well? There 
are at least two directions of generalization. 
The first direction is: are the findings valid 
for the subject of information systems in 
other countries. The second direction is: are 
the findings valid for other subjects? 

In order to answer both these directions of 
generalization more research is needed. 
Therefore, collecting additional data from 
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other subjects and other countries is pro¬ 
posed as future research. However, several 
of the problems identified and the solutions 
proposed are not formulated as being specif¬ 
ic Swedish or having a specific information 
systems character. 
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